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Abstract 


Successful  systems  for  classification  of  real-world  data  must  be  tolerant  of 
noise — that  is,  distortions  introduced  into  the  system's  model  of  the  real- 
world  domain.  Most  classification  systems  are  trained  on  a  set  of  exemplars 
to  identify  features  of  each  category  and  then  tested  on  previously  unseen 
instances.  In  an  instance-based  classification  system  using  l:-nearest  neigh¬ 
bor  (fc-NN),  the  training  phase  is  reduced  to  storing  one  or  more  exemplars 
for  each  category.  During  testing,  a  distance  metric  is  applied  to  the  features 
of  the  new  instance  to  determine  the  k  closest  exemplars.  A  voting  scheme 
assigns  the  category  of  the  modal  average  to  the  testing  instance.  Unlike 
other  methods,  fc-NN  does  not  try  to  distinguish  between  "relevant"  and 
"irrelevant"  features.  Nonetheless,  k-NN  has  been  shown  to  as5miptotically 
approach  optimal  Bayesian  accuracy. 

This  report  presents  the  results  of  appl5dng  k-NN  to  the  problem  of  classify¬ 
ing  chemical  agents  from  noisy  infrared  absorption  spectra  (from  a  suite  of 
chemical  agents  used  elsewhere  in  the  literature).  Straightforward  nearest- 
neighbor  approaches  without  editing  appear  to  be  tolerant  of  random  noise 
when  the  amounts  of  noise  in  the  training  and  testing  sets  are  relatively  close. 
Performance  of  k-NN  versus  1-NN  approaches  can  be  improved  if  the 
training  sets  are  edited  so  as  to  exclude  degenerate  outliers  and  redimdant 
positive  instances. 
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1.  Introduction 


The  problem  of  noise  or  xmcertainty  pervades  all  natural  classification  do¬ 
mains.  Noise  is  a  distortion  of  the  model  contained  in  the  classification  sys¬ 
tem  (as  defined  by  the  system's  representation  and  bias  and  as  indicated 
by  its  classification  performance)  from  the  observer's  or  experimenter's  re¬ 
ality.  Noise  in  a  classification  system  can  be  of  different  types,  arise  from 
different  sources,  and  be  introduced  at  different  points.  Successful  classifi¬ 
cation  of  real-world,  "field"  data  must  be  tolerant  of  all  these  t3^es  of 
noise.  In  this  report,  we  explore  the  effects  of  adding  normal  random  noise 
on  the  classification  of  chemical  agents  from  their  infrared  absorption 
spectra.  In  particular,  we  examine  the  classification  performance  of  an 
instance-based  classification  technique  known  as  k-nearest  neighbor 
(k-NN)  [1]. 

The  data  available  for  testing  consist  of  71  spectra  of  various  chemical 
agents.  The  data  were  collected  at  a  signal-to-noise  ratio  too  high  to  be  re¬ 
produced  outside  the  laboratory  environment.  (Field  data  will  have  much 
lower  signal-to-noise  ratios.)  Thus,  the  library  spectra  available  for  classifi¬ 
cation  purposes  are  "prototypical."  Of  particular  interest  is  how  accurate 
these  prototypical  data  are  for  classifying  noisy  data.  To  investigate  this 
point,  we  added  normal  random  noise  to  each  spectrum  to  create  noisy 
data  sets.  We  then  tested  the  classification  system  by  training  on  the  clean 
data  and  testing  on  the  noisy,  training  on  the  noisy  and  testing  on  the 
clean,  and  finally  by  both  training  and  testing  on  tiie  noisy  data.* 


2.  The  Problem  of  Noise 

Different  t)q5es  of  noise  include  "white  noise,"  "colored  noise,"  and  "clut¬ 
ter."  Gaussian  or  "white"  noise  is  uncorrelated  with  any  specific  values. 
"Colored  noise"  is  correlated  to  specific  values  contained  in  or  presented 
to  the  classification  system.  "Clutter"  results  from  environmental  condi¬ 
tions  outside  experimental  control.  Noise  may  exist  in  a  number  of  differ¬ 
ent  components  within  the  classification  system.  It  may  be  inherent  in  the 
classification  system  (probabilistic  Bayesian  belief  networks  or  probabilis¬ 
tic  search  algorithms,  for  example).  It  may  exist  in  the  limitations,  specifi¬ 
cations,  sensitivities,  faults,  etc,  of  the  system's  sensors  providing  the 
input.  Noise  may  exist  in  the  inputs  themselves  (because  of  variabilities 
imaccounted  for  in  the  environment,  incomplete  information — a  specific 
feature  not  currently  available,  for  example — or  incorrect  or  distorted  fea¬ 
ture  values).  It  may  be  introduced  during  the  processing  or  presentation  of 
the  inputs  (by  probabilistic  algorithms,  for  example).  Finally,  noise  may 
even  exist  as  the  misclassification  (either  malicious  or  accidental)  of  the  ex¬ 
emplars  used  for  training.  Any  natural  domain  will  inevitably  contain  all 
these  types  of  noise  to  varying  degrees.  Any  useful  classification  system 
will  have  to  be  tolerant  of  some  level  of  noise. 


*The  permutation  of  clean  training  and  clean  testing  was  not  feasible  to  this  problem  domain,  since  this  approach 
would  require  multiple  samples  of  every  category. 
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If  we  define  noise  as  a  distortion  of  the  reality  being  modeled,  then  the  bet¬ 
ter  the  classification  system  represents  the  domain  being  modeled,  the 
lower  the  intrinsic  noise.  For  all  practical  (as  opposed  to  philosophical) 
purposes,  the  noise  that  exists  in  the  inputs  to  the  system  and  that  intro¬ 
duced  by  the  system's  sensors  are  indistinguishable — they  both  result  in 
distortions  or  perturbations  to  the  values  of  the  features  or  attributes  pre¬ 
sented  to  the  system.  Noise  from  misclassification  of  the  training  exem¬ 
plars  is  discounted  by  some  researchers  as  "insufficient  to  characterize 
practical  situations"  [2].  In  fact,  if  we  consider  the  final  classification  as  just 
another  attribute  or  feature  of  the  input,  noise  of  this  type  is  just  as  preva¬ 
lent  as  the  other  types.  So  ultimately,  the  source  and  nature  of  the  noise  is 
irrelevant  to  the  classification  system.  Noise  intrinsic  to  the  classification 
system,  regardless  of  its  source  or  nature,  is  ultimately  what  is  to  be 
minimized. 

Researchers  in  machine  learning  have  long  recognized  the  need  for  noise 
tolerance  in  classification  systems  [3-10].  Most  of  these  researchers  under¬ 
stand  the  importance  of  noise  tolerance  but  fail  to  consider  it  in  their  own 
research.  Connectionist-based  approaches  using  artificial  neural  networks 
with  back  propagation  have  been  shown  to  be  tolerant  of  noise  [8,11]. 
Rauss  [11]  has  shown  that  in  this  problem  domain,  for  the  classification  of 
these  chemical  agents,  neural  networks  are  tolerant  of  noise  levels  as  high 
as  200  percent.*  Researchers  in  instance-based  and  decision-tree  methods 
claim  that  in  these  approaches  (unlike  connectionist-based  approaches)  ad¬ 
ditional  modifications  to  the  algorithms  are  necessary  to  handle  noisy  or 
uncertain  information  [4,5].  Some  instance-based  researchers  even  propose 
that  noisy  instances  (i.e.,  "outliers")  should  be  discarded  and  removed 
from  consideration  [12].  These  researchers  must  have  already  examined 
the  performance  of  these  systems  in  the  presence  of  noisy  or  uncertain  in¬ 
formation  and  found  it  to  be  inadequate.  These  results,  however,  are  not 
typically  presented.  This  report  quantifies  the  effects  of  different  levels  of 
normal  random  noise  on  the  accuracy  of  ^-NN  in  this  infrared  spectral 
identification  problem. 


3.  Problem  Domain 

The  domain  used  for  this  test  consisted  of  a  library  of  71  representative  ab¬ 
sorption  spectra  of  various  chemical  agents.  The  library  data  were  col¬ 
lected  in  a  controlled  laboratory  environment  at  very  high  signal-to-noise 
ratios.  The  spectra  thus  represent  "clean,"  "noise-free"  prototypes  for  each 
agent.  The  spectra  are  idealizations;  "live"  samples  from  an  actual  sensor 
in  an  xmcontroUed  envirorunent  that  happen  to  match  these  prototypes  are 
considered  merely  coincidental  and  not  expected.  Field  data  for  this  do¬ 
main  are  subject  to  numerous  uncontrollable  influences,  including  sensor 
limitations  (assumed  to  be  normally  distributed)  and  environmental 


*Since  Rauss's  method  of  introducing  noise  into  the  system  [11]  is  different  from  ours  in  these  experiments,  we  can¬ 
not  directly  compare  his  results  loith  ours.  His  paper  gives  details  on  his  approach. 
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conditions  (e.g.,  interference  from  other  chemical  agents).  All  these  factors 
complicate  the  problem  of  classifying  field  data  on  the  basis  of  the  library 
data. 

The  sensor  used  to  collect  these  data  was  a  Fourier  transform  infrared 
(FTIR)  spectrometer.  Each  spectrum  consists  of  571  features  corresponding 
to  the  infrared  absorption  rate  of  the  agent  at  wave  numbers  from  7.4  to 
12.5  pm — ^the  size  of  the  FTIR  window.  Each  spectrum  was  then  normal¬ 
ized  between  -1  and  1.  Rauss  [13]  has  shown  that  several  traditional  curve¬ 
matching  techniques  (absolute  value,  least-squares  fit.  Euclidean  distance, 
first  derivative  least-squares  fit,  and  least-squares  distance)  have  been  un¬ 
successful  in  classifying  the  agents  with  various  levels  of  noise.  But  Rauss 
was  examining  classification  accuracy  of  the  noisy  signals  by  comparison 
to  only  the  clean  library  spectra.  In  this  report,  we  empirically  quantify  the 
performance  of  the  fc-NN  algorithm  (using  Euclidean  distance)  when 
trained  and  tested  on  spectra  with  various  levels  of  noise  added. 


4.  Specification  and  Approach 

An  input  instance  to  the  system  is  specified  as  an  ordered  pair  consisting 
of  a  571-dimension  feature  vector  and  an  integer  value  denoting  the  correct 
target  classification  for  the  feature  vector: 

where  e  [0.0,  1.0]  is  the  normalized  value  of  the  spectra  at  the 
wavenumber,  and  e  [1,  71]  represents  the  correct  target  class.  During 
training,  the  correct  classification  is  made  available  to  the  classification 
system.  During  testing,  we  compare  the  correct  classification  to  the 
system's  classification  to  determine  classification  accuracy. 

The  classification  approach  tested  here  is  an  instance-based  method 
known  as  fc-nearest  neighbor  [13].  The  fc-NN  method  has  been  shown  to 
approach  Bayesian  classification  as  k  increases  [1].  We  implemented  the  k- 
NN  search  algorithm  using  k-d  trees  [14].  The  k-d  tree  algorithm  superim¬ 
poses  a  binary  tree  structure  over  the  571  features.  This  provides  for  re¬ 
trieval  of  the  m  closest  training  examples  in  expected  log2(N)  time  (where 
N  is  the  number  of  training  examples).  The  mechanism  for  balancing  the 
tree  lies  in  ranking  the  selection  of  which  feature  to  use  as  the  discriminant 
for  each  node.  The  binary  tree  structure  is  balanced  by  a  branch-and- 
bound  strategy,  which  selects  that  feature  for  branching  which  exhibits  the 
greatest  variance  over  all  the  remaining  instances  (i.e.,  instances  that  fall 
below  this  node  in  the  tree).  Thus,  the  feature  that  shows  the  greatest  vari¬ 
ance  over  all  training  instances  becomes  the  discriminant  feature  for  the 
tree's  root  node,  and  so  on.  The  partition  value  for  each  node  is  the  median 
value  of  the  node's  discriminant  feature.  The  search  for  the  m  training  in¬ 
stances  closest  to  a  testing  instance  then  proceeds  down  the  tree.  A  priority 
queue  of  the  m  closest  records  encountered  so  far  is  maintained  as  the 
search  progresses.  The  termination  test  is  crucial  for  expected  log2(N)  per¬ 
formance  when  m  is  greater  than  1.  Two  571-dimensional  tuples  are 
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maintained,  representing  the  lower  and  upper  bounds  for  the  features  vis¬ 
ited  so  far  in  the  current  subtree.  A  "ball"  (hypersphere)  around  the  testing 
instance  is  defined  with  radius  equal  to  the  distance  between  the  testing 
instance  and  the  closest  training  instance  encoimtered  so  far.  The 
search  can  be  terminated  whenever  the  ball  lies  entirely  within  either  the 
upper  or  lower  bound.  Backtracking  through  the  tree  is  necessary  only  if 
the  ball  aroxmd  the  testing  instance  intersects  the  bound  for  the  subtree 
opposite  the  current  one  (i.e.,  intersection  with  the  lower  boimd  if  the  ball 
is  centered  in  the  right  subtree  or  intersection  with  the  upper  boimd  if  the 
baU  is  centered  in  the  left  subtree). 

5.  Experimental  Design 

As  mentioned  earlier,  sources  of  noise  in  this  domain  result  from  sensor 
limitations,  environmental  conditions,  and  interference  from  the  presence 
of  other  chemical  agents.  Given  that  the  only  data  available  were  clean 
spectra,  the  problem  faced  here  is  to  add  simulated  noise  to  the  clean  li¬ 
brary  data.  In  this  report  we  simulate  noise  by  adding  various  levels  of 
white  noise  to  the  original  spectrum;  the  noise  is  distributed  around  a 
mean  of  0.0  with  variance  equal  to  the  level  of  noise  times  the  variance*  of 
the  original  spectrum.  We  then  renormalized  the  resultant  data  between  0 
and  1  by  finding  the  minimum  and  maximum  feature  values  of  the  new 
noisy  spectrum.  If  the  minimum  value  is  less  than  0,  the  difference  be¬ 
tween  0  and  the  minimum  value  is  added  to  all  the  feature  values.  Each 
feature  value  is  then  divided  by  the  maximum  value  (if  the  maximum 
value  exceeds  1).  We  also  normalized  the  original,  unnormalized  spectra 
(without  added  noise)  between  0  and  1  by  this  same  method  to  create  the 
clean  data  set.  Noise  was  added  according  to  this  definition  in  levels  of  5, 
10,  25, 50,  75, 100, 150,  and  200  percent  10  times  to  each  library  spectrum, 
resulting  in  noisy  data  sets  of  710  spectra  at  each  of  the  noise  levels.  Figure 
1  shows  a  sample  spectrum  with  the  various  levels  of  noise  added. 

5.1  Experiment  1:  Clean  Training/Noisy  Testing 

For  the  clean  training/ noisy  testing  experiment,  the  71  library  spectra  were 
used  for  training  purposes,  and  the  noisy  data  sets  (a  total  of  8  consisting 
of  710  noisy  spectra  each)  were  used  for  testing.  No  cross-validation 
method  could  be  used  in  this  experiment,  since  there  was  only  a  single 
training  instance  for  each  category.  For  the  same  reason,  only  the  closest 
matching  library  spectra  were  considered  (1-nearest  neighbor).  Table  1  and 
figure  2  summarize  the  results  of  dus  experiment. 


*Initially,  the  level  of  noise  used  was  a  percentage  of  the  standard  deviation.  But  it  was  noted  that  adding  10-percent 
noise  to  the  standard  deviation  was  equivalent  to  adding  only  1-percent  noise  to  the  variance.  The  variance  was  se¬ 
lected  because  the  resultant  amount  of  noise  was  greater  (until  the  noise  level  reached  100  percent). 
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Figure  1.  Sample 
spectrum  with 
various  levels  of 
noise. 


Table  1.  Clean 
training  (71 
samples)/noisy 
testing  (710  samples). 


8  10%  Noise 


Wave  No. 


100%  Noise 


Wave  No. 


75%  Noise 


Wave  No. 
200%  Noise 


Wave  No. 


Testing  noise  level 
(%) 

%  correct  classification 
(1-NN) 

5 

95.2 

10 

93.7 

25 

75.5 

50 

39.3 

75 

27.3 

100 

17.2 

150 

10.1 

200 

5.8 

5 


5.2  Experiment  2:  Noisy  Training/Clean  Testing 

For  the  noisy  training/ clean  testing  experiment,  the  71  clean  library  spec¬ 
tra  were  used  for  testing,  while  the  eight  noisy  data  sets  were  used  for 
training.  No  cross-validation  methods  applied  here,  for  the  same  reasons 
as  in  the  previous  experiment.  Unlike  experiment  1,  however,  more  than  1- 
nearest  neighbor  could  be  used,  since  the  training  set  consisted  of  multiple 
examples  (10)  for  each  category.  This  experiment  used  1-NN,  3-NN,  7-NN, 
and  17-NN.  For  k-NN  where  k  was  greater  than  1,  we  used  a  voting 
scheme  or  modal  average  to  detemiine  the  system's  classification.  (That  is, 
for  3-NN,  if  the  second  and  third  closest  training  instances  were  of  cat¬ 
egory  A,  while  the  first  closest  was  of  category  B,  the  system  would  clas¬ 
sify  the  test  instance  as  category  A.)  Ties  are  biased  in  favor  of  the  closest 
training  instance.  The  results  of  this  experiment  are  summarized  in  table  2 
and  figure  3. 

5.3  Experiment  3:  Noisy  Training/Noisy  Testing 

For  the  noisy  training /noisy  testing  experiment,  the  710  library  spectra  at 
each  noise  level  were  randomly  divided  (assuming  a  uniform  distribution) 
into  training  and  testing  data  sets.  The  training  data  sets  contained  639 
spectra,  while  the  test  sets  had  the  remaining  71  spectra.  This  division  was 
repeated  10  times  to  each  noisy  data  set,  and  tenfold  cross-validation  was 
used  to  calculate  the  average  classification  accuracies.  Each  training  data 
set  was  tested  against  every  test  set.  The  testing  used  1-NN  and  7-NN. 
Tables  3  and  4  and  figure  4  summarize  the  results  of  this  experiment. 
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Table  2.  Noisy 
training  (710 
samples)/dean 
testing  (71  samples). 


Figure  3.  Noisy 
training/clean  testing 
classification 
accuracy  (1-NN). 


Table  3.  Noisy 
training  (639 
samples)/noisy 
testing  (71  samples): 
l-nearest  neighbor. 


Table  4.  Noisy 
training  (639 
samples)/noisy 
testing  (71  samples): 
7-nearest  neighbor. 


Training 
noise  level 
(%) 

%  correct  classification 

l-NN 

3-NN 

7-NN 

17-NN 

5 

95.8 

95.8 

93.0 

90.1 

10 

84.5 

80.3 

71.8 

45.1 

25 

9.9 

8.5 

7.0 

9.9 

50 

1.4 

1.4 

2.8 

2.8 

75 

4.2 

4.2 

2.8 

4.2 

100 

4.2 

4.2 

2.8 

1.4 

150 

4.2 

2.8 

2.8 

2.8 

200 

2.8 

2.8 

2.8 

1.4 

Training  %  correct  classification  by  testing  noise  percentage 


noise  level 


(%) 

5% 

10% 

25% 

50% 

75% 

100% 

150% 

200% 

5 

93.2 

94.4 

88.9 

71.4 

52.8 

27.8 

15.1 

8.7 

10 

93.3 

95.2 

93.0 

77.9 

63.9 

38.0 

18.9 

12.5 

25 

79.7 

93.5 

92.5 

92.4 

84.7 

66.8 

38.9 

17.3 

50 

14.9 

31.7 

80.7 

93.9 

93.1 

86.8 

66.3 

39.6 

75 

9.8 

10.3 

47.9 

85.9 

94.8 

92.1 

84.4 

50.4 

100 

9.2 

6.6 

18.9 

70.3 

85.8 

92.3 

89.3 

73.9 

150 

8.7 

6.9 

8.3 

23.7 

46.5 

70.1 

93.7 

89.0 

200 

7.9 

6.5 

7.6 

11.0 

14.8 

29.3 

64.2 

90.1 

Training  %  correct  classification  by  testing  noise  percentage 

noise  level  - — - 


(%) 

5% 

10% 

25% 

50% 

75% 

100% 

150% 

200% 

5 

93.9 

94.7 

91.3 

70.0 

48.5 

26.6 

13.4 

8.9 

10 

91.1 

94.1 

91.3 

74.8 

58.2 

35.9 

18.6 

9.9 

25 

66.9 

86.9 

93.5 

89.0 

84.9 

58.6 

29.2 

15.8 

50 

14.7 
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Figure  4.  Noisy 
training/noisy 
testing  classification 
accuracy  (1-NN). 
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6.  Analysis  of  Results 

6.1  Experiment  1 

The  results  of  experiment  1  show  that  the  fc-NN  approach  trained  on  the 
clean  library  spectra  is  tolerant  of  low  levels  of  added  random  noise  (the 
approach  gave  93.7-percent  correct  results  in  testing  with  10-percent  added 
noise);  however,  performance  quickly  degrades  as  higher  levels  of  noise 
are  introduced.  Although  this  performance  degradation  was  not  linear,  it 
did  not  drop  off  nearly  as  fast  as  when  the  noisy  data  were  used  for  train¬ 
ing  and  the  clean  data  for  testing  (see  next  section). 

6.2  Experiment  2 

The  results  of  experiment  2  demonstrate  that  training  on  noisy  data  is  not 
effective  for  classifying  the  clean  library  spectra  if  the  noise  level  exceeds  a 
small  amount  (5  percent).  Performance  degradation  drops  off  quickly 
(nearly  a  step  function)  when  the  added  random  noise  exceeds  5  percent. 
An  interesting  observation  from  experiment  2  (borne  out  in  experiment  3) 
is  that  fc-NN  performed  increasingly  poorly  in  this  domain  as  more  neigh¬ 
bors  were  considered.  This  result  is  unexpected  in  light  of  the  fact  that  k- 
NN  approaches  Bayesian  classification  accuracy  as  k  increases  [1].  After 
examining  the  data,  we  determined  that  the  successive  degradation  in  per¬ 
formance  as  k  increased  was  generally  due  to  the  misclassification  of  more 
instances  of  the  same  chemical  agent.  For  example,  at  a  noise  level  of  5  per¬ 
cent,  1-NN  and  3-NN  made  the  same  three  mistakes:  one  instance  each  of 
agents  71,  57,  and  60  was  misclassified.  7-NN  correctly  classified  the  in¬ 
stance  of  agent  57,  but  failed  to  correctly  classify  an  instance  of  agent  24 
and  two  additional  instances  of  agent  60.  17-NN  misclassified  an  addi¬ 
tional  instance  of  agent  24,  as  well  as  two  new  instances  of  agent  57  (17- 
NN  did,  however,  correctly  classify  the  two  instances  of  agent  60 
misclassified  by  7-NN). 

This  behavior  seems  to  suggest  that  the  accuracy  of  fc-NN  for  larger  values 
of  k  depends  on  the  differences  between  positive  training  instances  of  the 
same  category.  All  the  instances  used  for  training  in  this  data  set  carried 
essentially  the  same  information.  They  were  merely  copies  of  one  another 
with  randomly  generated  differences.  Since  the  amoimt  of  additional  in¬ 
formation  available  in  successive  copies  is  minimal  and  accidental,  the 
benefit  of  retaining  those  additional  copies  as  training  instances  was  out¬ 
weighed  by  the  fact  that  the  random  variations  in  some  of  the  copies 
resulted  in  degenerate,  "outlier"  instances  that  are  closer  to  outliers  of  an¬ 
other  class  than  to  their  own  (fig.  5).  It  is  not  the  number  of  positive 
instances  of  a  particular  class  that  is  important  for  nearest-neighbor  classi¬ 
fication.  Rather,  it  is  how  representative  each  positive  instance  is  of  the 
class.  Additional  positive  instances  are  potentially  detrimental  unless  they 
represent  different  characteristics  or  aspects  of  the  class  not  contained  in 
other  positive  instances.  This  result  suggests  that  the  training  data  used  in 
these  tests  should  be  subjected  to  some  form  of  editing. 
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Figure  5.  Pictoral 
explanation  of  ic*NN 
phenomenon 
observed  in 
experiments  2  and  3. 


One  form  of  editing,  suggested  by  Wilson  [1],  is  to  eliminate  any  training 
instances  that  are  not  classified  correctly  (as  determined  by  the  fc-NN  on 
the  remaining  instances).  This  procedure  would  result  in  degenerate  out¬ 
liers  being  eliminated  from  the  training  set.  Alternatively,  one  could  elimi¬ 
nate  all  but  one  of  the  training  instances  that  are  classified  correctly.  This 
procedure  could  result  in  redundant  training  instances  being  eliminated. 
Yet  another  approach,  suggested  by  Cost  and  Salzberg  [12],  is  to  leave  the 
degenerate  or  redundant  instances  in  the  training  set  but  define  "exception 
spaces"  around  them  by  weighting  the  conditional  probabilities  of  their 
feature  space*  according  to  the  number  of  other  exemplars  closest  to  them 
but  in  different  categories.  This  approach  would  result  in  degenerate  outli¬ 
ers  having  less  attractive  power  (i.e.,  it  limits  their  "sphere  of  influence") 
and  effectively  distances  them  further  from  other  "confusables"  close  to 
them  in  the  feature  space  but  with  different  classification  categories. 
Regardless  of  the  approach,  some  form  of  editing  should  be  used,  since  the 
results  of  this  experiment  suggest  that  indiscriminate  inclusion  of  degener¬ 
ate  or  redundant  instances  in  the  training  set  can  result  in  decreased  classi¬ 
fication  accuracy. 

6.3  Experiment  3 

In  experiment  3,  every  noisy  test  data  set  was  tested  against  every  noisy 
training  data  set.  The  idea  behind  this  experiment  was  to  see  if  any  par¬ 
ticular  noise  levels  were  better  or  worse  for  training.  Although  we  cannot 
say  that  any  of  the  training  sets  proved  especially  accurate  relative  to  the 
others,'*’  it  is  interesting  not  only  that  relatively  high  levels  of  accuracy 


*Cost  and  Salzberg  use  a  distance  metric  known  as  “value  difference  metric"  (VDM),  based  on  the  conditional  prob¬ 
ability  (using  a  relative  frequency  interpretation)  of  a  specific  category  given  a  value  for  a  dimension  in  the  feature 
space  (i.e.,  p(category  \  feature)). 

performed  an  F  test  to  test  the  null  hypothesis  that  the  classification  accuracy  was  not  affected  by  the  differences 
in  noise  levels  in  the  experiments  in  which  the  same  noise  level  was  used  for  training  and  testing  (the  diagonal  of  the 
matrix  in  table  3).  Wie  accomplished  this  by  comparing  the  sum  of  squares  between  the  groups  fSj,)  to  the  sum  of 
squares  within  the  groups  (S^J,  where  the  ^oup  is  defined  as  the  percentage  of  noise  added  before  training/testing.  Let 
Yjj  represent  the  i*^  observation  from  thef'  group: 
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were  obtained  on  the  test  data  sets  that  contained  the  same  amount  of 
noise  as  the  training  set  (which  is  to  be  expected),  but  also  that  the  noisy 
training  data  sets  seemed  to  be  tolerant  of  a  wider  range  of  noise  (and  their 
performance  degradation  closer  to  linear)  than  the  clean  library  spectra 
training  set  (fig.  5).  This  suggests  that  it  may  be  worthwhile  to  determine  a 
ballpark  estimate  of  the  amount  of  noise  that  will  be  encountered  in  the 
field  and  train  on  instances  with  that  amoimt  of  added  noise.  Even  if  the 
ballpark  estimate  is  not  accurate,  if  it  is  relatively  close,  classification  accu¬ 
racy  might  be  increased. 


7.  Conclusions 

The  experiments  performed  here  suggest  that  straightforward  nearest- 
neighbor  approaches  without  editing  are  tolerant  of  random  noise  when 
the  amoimt  of  noise  in  the  training  set  is  relatively  close  to  the  amount  of 
noise  in  the  test  set.  Experiment  3,  in  which  the  classification  system  was 
both  trained  and  tested  on  noisy  data,  demonstrated  better  performance 
with  greater  tolerance  of  different  levels  of  noise  than  the  results  obtained 
by  Rauss  [13].  Again  however,  Rauss  compared  the  performance  of  testing 
on  noisy  data  only  after  training  on  clean,  "noise-free"  data  (i.e.,  experi¬ 
ment  2  here).  We  conclude  that  careful  consideration  of  the  training  set 
relative  to  the  testing  environment  can  result  in  substantially  improved 
performance  of  nearest-neighbor  classification  techniques.  The  best  results 
are  obtained  when  the  training  and  testing  sets  have  roughly  the  same  lev¬ 
els  of  noise.  When  the  amount  of  noise  in  the  training  set  is  drastically  dif¬ 
ferent  from  that  in  the  test  set,  performance  degradation  was  observed. 

Finally,  the  poor  performance  of  k-NN  compared  to  1-NN  indicates  that 
additional  copies  of  essentially  the  same  positive  instance  can  degrade 
classification  accuracy.  Some  form  of  editing  should  be  done  to  reduce  the 
effect  of  degenerate  outliers  and  redundant  data. 


and 

S„  =  ZniX{'^r^f  ■ 

Based  on  the  1-NN  classification  results,  =  351.99  with  71  degrees  of  freedom  (80  tests  -  8  groups  - 1),  and  Sj,  = 
91.95  with  7  degrees  of  freedom  (8  groups  - 1).  The  F  ratio, 

1^x9  =  2.6893  , 

is  compared  to  the  95-percent  confidence  level  ofF(7, 72)  =  2.13966.  Since  the  F  ratio  is  less  than  the  95-percent  con¬ 
fidence  level  ofF(7,  72),  we  reject  the  null  hypothesis  that  classification  is  independent  of  the  training/testing  noise 
level  with  95-percent  confidence. 
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8.  Further  Research 


Further  experiments  need  to  be  nm  with  an  edited  nearest-neighbor  tech¬ 
nique.  Suggested  further  experiments  include  (1)  editing  the  training  set 
by  throwing  out  degenerate  outliers  that  are  misclassified  by  the  other 
positive  instances  of  the  class,  (2)  editing  the  training  set  by  throwing  out 
redimdant  positive  instances  of  the  class,  and  (3)  identifying  degenerate 
positive  instances  that  misclassify  other  positive  instances  and  weighting 
them  proportionally  to  their  misclassification  accuracy,  so  as  to  distance 
them  further  from  other  training  instances. 

We  also  want  to  examine  and  compare  other  classification  techniques  on 
this  same  problem  domain  and  data  sets.  Other  methods  warranting  con¬ 
sideration  are  connectionist-based  approaches  (neural  networks),  axis- 
parallel  [4]  and  oblique  decision  trees  [15],  and  genetic  algorithms  [16]. 

Additionally,  a  problem  of  great  interest  in  this  domain  is  the  classification 
of  multiple  chemical  agents  present  in  the  same  spectrum.  This  is  the  ex¬ 
ample  of  "clutter"  mentioned  earlier,  where  a  mixture  of  agents  in  the 
same  spectrum  causes  an  interference  problem  in  the  identification  of  each 
individual  agent.  To  test  the  classification  system's  performance  on  this 
mixture  problem,  one  would  generate  test  data  sets  by  randomly  selecting 
two  different  agents  and  adding  their  spectral  contents.  The  training  sets 
would  remain  as  presented.  With  2-NN  or  greater  used,  the  system  would 
be  penalized  most  heavily  when  neither  of  the  agents  appeared  in  the 
k-closest  list,  penalized  less  if  at  least  one  of  the  agents  appeared  in  the 
k-closest  list,  and  not  penalized  at  all  if  both  the  agents  were  among  the 
k-closest. 

Finally,  a  potentially  worthwhile  approach  that  fuses  connectionist-based 
and  instance-based  classification  techniques  would  be  to  create  a  separate 
perceptron  for  each  category.  Test  instances  would  then  be  fed  to  each 
perceptron,  and  the  activation  levels  of  the  separate  perceptrons  would 
constitute  the  "distance"  (actually  the  "distance"  would  be  defined  as  1.0  - 
activation  level)  between  the  category  and  the  test  instance.  The  fc-NN 
technique  could  then  be  applied  to  find  the  closest  classifications. 
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